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Volume I: Technical Report 

1.0 Authorization and Notification 

Mr. Doug Cooke, NASA Deputy Associate Administrator for the Exploration Systems Mission 
Directorate (ESMD) at NASA Headquarters, requested a Crew Exploration Vehicle (CEV) 
Avionics Integrated Lab (CAIL) Independent Analysis to compare the strategic differences, risks 
and benefits between the current CAIL approaches to alternative NASA approaches. The NESC 
team worked independent of the Constellation Program (CxP) to evaluate both flight hardware 
and software development. 

-The NESC team started their comparison with the Shuttle Avionics Integration Laboratory 
(SAIL) and robotic mission “Flat-Sat” approach. 

An NESC out-of-board activity was approved February 8, 2008 and Mr. Mitchell L. Davis, 
NASA Technical Fellow for Avionics, was chosen to lead this assessment. A stakeholder 
outbrief was presented to NESC Review Board (NRB) on March 13, 2008. A formal stakeholder 
outbrief was presented to Dr. Richard Gilbrech, Associate Administrator for the Exploration 
Systems Mission Directorate (ESMD) at NASA Headquarters, and guests at NASA Headquarters 
on March 18, 2008. The final NESC CAIL report was presented to the NRB on May 1, 2008. 

The key stakeholder for this assessment is Mr. Doug Cooke. The project’s point of contact is 
Mr. Aaron Van Baalen, Avionics and Software Test &Verification (T&V) Lead, Orion Project 
Office. 
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4.0 Executive Summary 

Two approaches were compared to the Crew Exploration Vehicle (CEV) Avionics hitegration 
Laboratory (CAIL) approach: the Flat-Sat and Shuttle Avionics Integration Laboratory (SAIL). 
The Flat-Sat and CAIL/SAIL approaches are two different tools designed to mitigate different 
risks. Flat-Sat approach is designed to develop a mission concept into a flight avionics system 
and associated ground controller. The SAIL approach is designed to aid in the flight readiness 
verification of the flight avionics system. The approaches are complimentary in addressing both 
the system development risks and mission verification risks. 

The CAIL concept is modeled after the SAIL approach with a number of notable differences. 
These include: For the Orion Project, a coordinated, end-to-end box to system level development 
and verification process was not evident in the information provided. For the CAIL concept, a 
coordinated, end-to-end box to system level development and verification process was not 
evident in the information provided. The CAIL concept assumes the flight subsystems will be 
matured for the system level verification prior integration into the CAIL. It was not apparent to 
the NESC team that there was a documented plan to assure unit/sub-system maturity prior to 
installation into the CAIL. Both the CAIL and the Lockheed-Martin (LM) operated Exploration 
Development Laboratory (EDL) documents did not reference any intentional effort to provide an 
engineering team knowledge transfer from the CAIL integration and verification team to the 
ground operations and mission operations teams. The NESC team noted that, based on their 
experience, successful projects are keenly aware of their tools’ abilities and limitations and can 
quickly adjust to the dynamically changing risk priority and hardware/software configurations 
during the development phase. Coordination of the verification process is essential to minimize 
the risk of missing key issues and to be prepared for schedule deviations. 

The Flat-Sat approach starts early in the development and is designed to mature into an accurate 
representation of the flight system. It then transitions to supporting flight software development 
and mission operations. The SAIL approach is a high fidelity engineering replication of the 
flight article designed to verify integrated system performance, capture system discrepancies 
from mission to mission, and aid in reducing schedule pressures on the flight article. 

The SAIL approach is to use flight qualifiable or qualified hardware in the flight systems. 
However, depending on unique test requirements, it may use component simulators or 
engineering test units (ETUs). 

The following NESC team findings were identified: 

• A coordinated, end-to-end, box to system level development and verification process was 
not evident for the proposed CAIL concept. 
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• The CAIL assumption is that the flight subsystems will be matured for the system level 
verification. 

• The Flat-Sat and SAIL approaches are two different tools designed to mitigate different 
risks. 

The following NESC team recommendation was provided: 

• Define, document, and manage a detailed interface between the design and development 
(EDL and other integration labs) to the verification laboratory (CAIL). 

The Orion Project Office is actively working this recommendation. 


NESC Request No.: 08-00451 


© 

NASA Engineering and Safety Center 
Report 

Document #: 

RP-08-31 

Version: 

1.0 

Title: 

Crew Exploration Vehicle (CEV) Avionics Integration 
Laboratory (CAIL) Independent Analysis 

Page #: 
10 of 39 


5.0 Task Overview 

To accomplish this quick turnaround assessment (3 -week timeframe), the approach taken 
involved a combination of interviews of subject matter experts (SMEs) associated with Flat-Sat 
and SAIL operations and a review of available documentation associated with the proposed 
Orion CAIL concept. Section 10.0 provides the references the NESC team reviewed for this 
assessment. 

The SME interview process started with the Goddard Space Flight Center (GSFC) Integration 
and Test (I&T) Branch Head was interviewed for a detailed summary of the Flat-Sat approach. 
This summary was shared with the NESC team with additional inputs from: Victor Mora, Jet 
Propulsion Laboratory (JPL); the NASA Technical Fellows for Avionics and Software; and Bill 
Ritz, retired SAIL Manager. Mr. Ritz, retired SAIL Manager, provided the team with a detailed 
overview of the SAIL operation and development history. Aaron Van Baalen, Avionics and 
Software T&V Lead, provided a detailed description of the planned CAIL operations approach. 

On March 3, 2008, the Johnson Space Center (JSC) Engineering Directorate, Avionic Systems 
Division, senior leaders and CAIL team leaders met at GSFC for a Flat-Sat discussion and tour 
of the Lunar Reconnaissance Orbiter (LRO) (Figure 5.0-1), Solar Dynamics Observatory (SDO) 
(Figure 5.0-2), Tropical Rainfall Measuring Mission (TRMM) (Figure 5.0-3), and Wilkinson 
Microwave Anisotropy Probe (WMAP) Flat-Sat laboratories. Additionally, the Hubble Space 
Telescope’s Vehicle Electrical System Test (VEST), which serves a similar purpose as the 
SAIL/CAIL (Figure 5.0-4), was also visited. At the conclusion of this visit, good understanding 
on the similarities and differences of the various approaches among the NESC team, JSC 
Engineering Directorate senior leaders, and the CAIL team leaders was achieved. 



Figure 5.0-1. Lunar Reconnaissance Orbiter Breadboard (left), and Lunar Reconnaissance 

Orbiter Spacecraft (right) 


NESC Request No.: 08-00451 



@ 

NASA Engineering and Safety Center 
Report 

Document #: 

RP-08-31 

Version: 

1.0 

Title: 

Crew Exploration Vehicle (CEV) Avionics Integration 
Laboratory (CAIL) Independent Analysis 

Page #: 
11 of 39 



Figure 5.0-2. Solar Dynamics Observatory Flat-Sat Engineering Test Units (left) and 

Ground Controller (right) 



Figure 5.0-3. Tropical Rainfall Measuring Mission Flat-Sat in Operational Support 
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Figure 5.0-4. Hubble Space Telescope VEST in Clean Room 


5.1 Integration and Test Process 

For this report, I&T is the process that matures a design concept and develops and integrates the 
flight hardware with ground controller systems into a fully integrated operational system. The 
NESC team found that some terms used to describe the I&T process caused confusion during 
interchanges due to location specific definitions. Therefore, for this assessment, the NESC team 
clarified the I&T definition to enable focusing on the general process and not the specific 
different details. 

• “Development” refers to the process of functional testing, interface testing, open-loop 
testing, and close-loop testing necessary to mature a concept into an operational system. 

• “Verification” refers to the fonnal process of demonstrating a design is flight ready and is 
assumed to be interchangeable in the general sense with the terms verification, validation 
and certification. 

• “Flight Avionics System” refers to all electrical/elcctronic systems that will fly on the 
vehicle. 

• “Ground Controller” refers to the ground electrical/electronic systems, tools, and 
procedures that enable the development and operation of the Flight Avionics System. 

During the development process, different combinations of electrical/elcctronic systems and 
ground systems replicate various flight configurations enabling the design evolution from 
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concept into a flight-ready system. There is no standard and fixed tool set; rather the tools 
depend directly on the project’s risks and mitigation plan. Some examples of these tools include 
software test beds, computer development unit, and simulators. The NESC team noted that, 
based on their experience, the successful projects are the ones who are keenly aware of their 
tools’ abilities and limitations and can quickly adjust to the dynamically changing risk priority 
and hardware/software configurations of the development phase. 

Early in the development cycle of a space flight design, high level requirements are parceled out 
to lower level subsystems, as shown in Figure 5.1-1. The requirements are then translated into 
functions that will be implemented in the specific hardware or software subsystem. Decisions 
are made regarding all aspects of the design, from the electrical circuit architecture to wire 
routing. These decisions will then define the design features. The subsystem is assembled into a 
system with an associated ground controller. Later in the development process, the subsystem 
was confirmed that the initial requirements were met and, more importantly, all other design 
features were documented. The requirements are the minimal set of functions that the system 
must do where the design features are all functions the system can do. 


Sub-system 

Allocation 



(Requirements 


Software 
Functions ' 


Sub-system 

Design 


Design 
Implementation N* 


Design 

’ Implementation 


Flight System 
Design 

Intended 
Functions With 
Other Features 



Verify intended requirements 
AND 

characterize other desirable 
features 


Characterize benign features 
w / perhaps operational 
constraint or limitation 


Remove unacceptable 
features from design 


time 

Figure 5.1-1. Flight System Development Process 


A large portion of these design features are desirable and aid in meeting the requirement. For 
example, the power interface and the command interface to a computer which implements the 
control requirement are necessary design features (e.g. results of the implementation). Some of 
these design features are benign and may not be desirable, but are not at a risk level that would 
dictate removal from the design; rather they lead to a limitation or operational constraint. For 
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example, a condition in which simultaneously executing commands subsystem-A ON and 
subsystem-B ON under worst case conditions will trip the circuit breaker. Since the ON 
commands are an infrequent operation, one is willing to accept an operational constraint of 200 
milliseconds between commands. The remaining design features are those that are undesirable 
and can lead to a premature anomaly or system failure. Once identified, the undesired feature is 
removed from the design. The undesirable features are discovered throughout the development 
phase by identifying a discrepancy between a predicted or expected subsystem response and the 
actual hardware/software response. 

The Flat-Sat approach is commonly used in the robotic spacecraft development where a single or 
limited number spacecraft or set of spacecraft are built. For the development of the single 
spacecraft, the fonnal system verification must occur on the flight unit to contain the risk to an 
acceptable level. The Flat-Sat serves three general purposes which align to the different phases 
of the development: I&T development (primary purpose), the flight software development 
(secondary purpose), and the flight operations support and maintenance (tertiary purpose). The 
Flat-Sat approach is optimized for development where the fidelity and configuration control is 
moderated to enable rapid design evolution from an initial concept to a flight ready system. For 
the single or set of missions, the formal verification process occurs on the flight unit and 
therefore there is less need for flight readiness verification on a high fidelity copy. Figure 5.1-2 
compares the Flat-Sat approach to the SAIL/CAIL approach. 
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System 
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Figure 5.1-2. Comparison of the Flat-Sat Approach to the SAIL/CAIL Approach 


For the SAIL approach, high fidelity and rigorous configuration control enables flight avionics 
verification with high degree of confidence for the first mission and subsequent missions. The 
SAIL approach is optimized to identify subtle discrepancies in the design by a high fidelity 
replication of the flight avionics system. For multi-missions, the SAIL approach provides the 
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tool for the formal flight readiness verification prior to final installation on the vehicle and 
allows identification and resolution of system level discrepancies from mission to mission. 

5.2 Flat-Sat Approach 

The Flat-Sat begins assembly shortly after the Preliminary Design Review (PDR). ft is an 
assembly of key subsystem Engineering Test Units (ETUs), simulators, ground controller and a 
flight-like harness. A ground controlling system is developed in conjunction with the flight-like 
avionics system. Shortly after the Critical Design Review (CDR), the amount of testing on the 
Flat-Sat accelerates as the spacecraft matures. When a major subsystems arrives, extensive 
interface and functional tests are completed which result in discovering discrepancies in the 
requirements or design that lead to improvements in the flight design and increased accuracy of 
the Flat-Sat’s simulation. Discovering these discrepancies in this early phase allows for changes 
to be incorporated into the flight design prior to delivery. Since the design engineers and the 
ground operations team are both working concurrently on the Flat-Sat, there is beneficial 
knowledge transferred between the teams. The Flat-Sat approach also enables the concurrent 
development of all integration plans and procedures, the ground controller command files, and 
the telemetry files data base. 

The GSFC practice is to not place a flight unit into the Flat-Sat mainly due to the lack of 
historical pedigree of the associated flight-like avionics. However, JPL does not have this 
practice. It is common to place a Flat-Sat ETU into the flight subsystem as a place-holder while 
the flight unit is elsewhere. Around the time the first Comprehensive Performance Test is 
completed on the flight system, the Flat-Sat would primarily be supporting flight software 
development. When a discrepancy is discovered that results in a hardware change, the corrective 
action is demonstrated on the Flat-Sat before implementation on the flight system. Upon launch, 
the Flat-Sat operation is transferred to the flight operations team for flight operational support 
and software maintenance. Figure 5. 2-1 illustrates this progression. For large out-of-house 
spacecraft like Earth Observing System (EOS)-Aqua and EOS-Aura, a Flat-Sat is a contractor- 
owned system, with a separate, nearly identical Flat-Sat system delivered for operational support. 
See Figure 5.2-1. 
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Figure 5.2-1. Flat-Sat Approach Phasing Relative to the Flight System 


5.2.1 Flat-Sat System Fidelity 

The Flat-Sat hardware is comprised of ETUs that are modified to be configurationally identical 
with the current flight design. Additionally, fidelity of the simulators increase as discrepancies 
are discovered and mitigated by design modifications. The fidelity of the wire will vary 
depending on the potential risk. High speed complex interfaces require duplication of the flight 
design and length. Experience has shown that a lower fidelity power harness is an acceptable 
risk and thus the exact flight length is not typically replicated. Figure 5.2-2 describes, in a 
notational sense, variations in fidelity of the Flat-Sat over time. The discrepancies in the ordinate 
axis are defined as system responses that are different from expected responses. As mentioned 
earlier, the discrepancies are a result of design features which can be beneficial, benign or 
undesirable. It is the benign design features which lead to limitation or operational constraints 
and the undesirable design features which are most critical to identify. These hidden 
discrepancies are discovered by operational time and testing of the Flat-Sat system. Note that a 
finite or limited time from initial concept to launch, which is what creates the acceptable risk 
level. 
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Figure 5.2-2. Notional Diagram of Flat-Sat range of Fidelity 


The Flat-Sat approach is a risk trade bounded by two limits: insufficient fidelity and insufficient 
time. Both impede the ability to identify discrepancies prior to entering the flight system 
verification phase. In the flight system verification phase, a design change halts the verification, 
the hardware is corrected, and time consuming regression testing is required. Therefore, the 
desire is to maximize the discovery of (benign and undesirable) discrepancies prior to the 
verification phase. At one extreme, a Flat-Sat placed in operation too quickly will be of 
insufficient fidelity and will not adequately simulate the flight system, thus the rate of 
discovering discrepancies will result in the detection of an unacceptable number of discrepancies 
upon entering flight system verification. At the other extreme, building an exceptionally high 
fidelity Flat-Sat consumes a significant portion of the available time and impacts configuration 
management complexity. It also becomes cumbersome to reconfigure Flat-Sat systems resulting 
in insufficient time to identify and resolve discrepancies prior to entering the flight system 
verification phase. The goal is to verily all requirements on the flight system. However, those 
requirements that can not be verified on the flight system are verified on the lower fidelity Flat- 
Sat. The risk associated with a lower fidelity Flat-Sat is not directly translated to higher mission 
risks. The reason is most requirements are verified on the flight system, and the requirements 
verified on the Flat-Sat are analyzed to be at the appropriate fidelity. 

Hardware and software test automation requires command and control capabilities above and 
beyond flight system operations. External control of test simulators and hardware is required to 
set initial conditions and repeat the test from these initial conditions. The capability to sense, test, 
and record the system response is required for automated test execution and pass-fail evaluation. 
Off-nominal behavior requires insertion or simulation of system faults in hardware and software, 
under control of the test scripts. System performance can be evaluated within the margins and 
limitations of the test environment fidelity. High fidelity portions of the system can be evaluated 
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for performance, even within an overall low-fidelity test system. Within the Flat-Sat 
environment, knowledge transfer of ground and mission operations is value-added to the 
operation, test, and development personnel. Familiarity of the mission operations equipment, 
procedures, and support software optimizes the quality of this knowledge transfer. 

5.2.2 System Software Development Major Attributes of Integration Labs 

The scope and needs of the laboratory customers directly influence the test bed fidelity. Test bed 
customers include simulation developers, hardware and software integration and test engineers, 
ground operations and ground application developers, mission operators, external hardware 
interface engineers and system integration engineers. Scheduling the use of the system, with 
margin for delivery and failure analysis, detennines the required number and capabilities of the 
test beds. 

Simulation developers require access to high fidelity interfaces and models to identify the 
simulation accuracy and limitations of flight and ground systems. The command and control 
interface to these simulations need to be implemented and supported. Hardware and software 
integration and verification requires test suite development, command and control interface 
development, and access to interface signals for diagnostic purposes. Ground operations require 
test suite development time for enviromnent and integration tests, test control and telemetry 
monitoring, and code modification procedures. External users will need access to the test bed for 
acceptance testing. 

Sparing of hardware for the test bed and the time expected to repair the test bed should be 
considered. This drives maintenance personnel and maintenance scheduling requirements. 

5.2.3 Software Development Parallel Analysis, Evaluation, Integration, and Test 

Multiple test beds of varied fidelity can be used in parallel to support user demands. An effort 
must be made to detail what subsystems are verified on which test environment. Software 
partitioning the system into subsystems, with the minimal interfaces and the minimal 
dependencies, reduces integration risk and effort when the subsystems are integrated. 

Maximizing the similarity of the test operations interfaces to the actual ground and flight 
operations interfaces provides an early start for ground and operations development and 
knowledge sharing between personnel. Even a low-fidelity test environment can provide early 
insight into development issues. 

Software and software test bed configuration management ranges from minimal early in 
development to rigorous enforcement as the subsystem test beds migrate toward high-fidelity 
system simulations. The configuration management complexity is directly related to the number 
and variety of uses the test bed is required to support. Constantly switching between 
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configurations to support differing users needs to be addressed in the configuration management 
plan. 

Customers for a test bed can include parties external to a project, with conflicting schedules and 
priorities. Non-NASA-owned test beds can create conflicts and competition over schedule, 
configuration, and access to the use of the test bed. 

5.2.4 Flat-Sat Summary 

The Flat-Sat approach regulates the hardware/software and test bed configuration control process 
to a level that enables rapid development of the hardware and software and, as the design 
matures, the fidelity and configuration control increases appropriately. Mission success is 
directly linked to the earliest possible testing which enables corrective feedback into flight 
hardware/software design prior to fabrication. The I&T process and procedures development 
along with Flat-Sat is another advantage, which reduces the risk of inadvertent mistakes on the 
flight system. Finally, developing the Flat-Sat in conjunction with the eventual ground controller 
allows a solid knowledge transfer between the design engineering and mission operations teams. 
Although the lower fidelity system will allow early discrepancies to escape, past experience has 
shown that overall this is the optimal approach for development and balancing cost and schedule. 

5.3 SAIL Approach 

Relative to Shuttle Avionics Integration Laboratory (SAIL) background, the Space Shuttle Task 
Group and technology program planning in the 1960/1970 time frame included simulation 
testing of an integrated electronics system to ensure compatibility of individual systems. The 
intent would be verification of functions involving several subsystems perfonning as a whole, 
and would lead to development of operational procedures (human/machine functional 
relationships, etc.) and software design and test. Completeness of the simulations would be such 
that they are faithful analogs of the mission/configuration under test or investigation. 

The Space Shuttle Program (SSP) planning included a systems integration laboratory for 
subsystem development and integration, flight control, and mission verification of flight 
hardware/software (e.g. avionics). All Space Shuttle contractor proposals included 
subsystems/systems integration laboratories. The SAIL was located at the Johnson Space Center 
(JSC) to make use of: existing Government facilities; JSC civil service and support contractor 
expertise; proximity to the software development laboratory; ability to support Shuttle 
operational phase without being committed to a prime contractor (i.e., reduced operational 
costs); and better NASA insight into the development of a computer-controlled, integrated 
avionics system — a novel idea in the 1971 time frame. For example, the facility and test support 
systems (integrated test area, test operations center, dynamics simulation, etc.) were provided by 
JSC. 
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The SAIL was a multi-center, multi-system that serviced three separate centers for vehicle I&T 
and operations. The Orbiter flight system and pertinent Ground Support Equipment (GSE) was 
provided by the Space Shuttle prime/integrating contractor. The Marshall Mated Elements 
System (MMES) was supplied by the Marshall Space Flight Center (MSFC) after the Approach 
and Landing Test (ALT), and included flight system components and pertinent GSE and test 
support/simulation systems. The Launch Processing System (LPS) was provided by the Kennedy 
Space Center (KSC). 

The SAIL is used as an integration and verification laboratory. System level integration was 
completed in the SAIL before formal verification. As new elements or design changes are 
implemented on the Space Shuttle, the SAIL is utilized for integration and test prior to formal 
verification. The Phase I breadboard SAIL was activated in the 1973 time frame. This allowed 
early installation of laboratory test systems, integration of early flight subsystems, and 
accomplishment of laboratory performance testing. Major SSP activities supported by SAIL 
included Phase II ALT, Phase III Orbital Flight Test (OFT), and Phase IV Mission Support. A 
significant activity between the completion of Phase II ALT and start of Phase III OFT was the 
Operational Flight Test Changeover which involved complete replacement of the avionics, bays, 
and wire harnesses. 

Other significant support activities included: LPS I&T with the SAIL flight system, pre- 
acceptance test of the Remote Manipulator System (RMS) (prior to delivery of the first "arm"), 
Ku-band subsystem integration and test, and multi-function electronic display system 
I&T. During the three phases of SAIL subsequent to Phase I, two test stations were utilized. The 
Space Shuttle test station was configured with flight qualifiable or flight- like avionics hardware 
in a multi-string configuration. This hardware was in a three-dimensional mock-up consisting of 
a cockpit, forward equipment bays, and aft equipment bays. Wire harnesses for this mock-up 
were flight configuration (length, size, material, and routing) except where SAIL facility 
limitations dictate otherwise. Figure 5.3-1 illustrates the vehicle layout in the SAIL Shuttle Test 
Station. The SAIL Shuttle Test Station Orbiter Flight System was designated Orbiter Vehicle 
(OV)-095. The Guidance, Navigation, and Control (GN&C) test station utilizes flight-type 
hardware mounted in racks connected with non-flight cabling. The test station supported GN&C 
ascent and limited on-orbit verification testing. The need for the GN&C test station was due to 
the overall verification test workload. The GN&C test station was decommissioned in the 1983 
time frame and recommissioned in the 1993 time frame, and later decommissioned, due to 
changes in SSP requirements. Figure 5.3-2 illustrates the SAIL Mission Capability. 
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Figure 5.3-1. SAIL Shuttle Test Station Vehicle Layout 


NESC Request No.: 08-00451 


@ 

NASA Engineering and Safety Center 
Report 

Document #: 

RP-08-31 

Version: 

1.0 

Title: 

Crew Exploration Vehicle (CEV) Avionics Integration 
Laboratory (CAIL) Independent Analysis 

Page #: 
22 of 39 


EXTERNAL TANK 
SEPARATION 



Figure 5.3-2. SAIL Mission Capability 


Overall, the SAIL is a central facility where avionics and related flight hardware (or simulations 
of this hardware), flight software, flight procedures, and associated GSE are brought together for 
integration and mission verification testing. The SAIL supports avionics integration, test, and 
verification from pre-launch to abort phases such as Return-To-Launch Site (RTLS), and landing 
and roll-out. The SAIL configuration supports total integration of the Orbiter avionics, Shuttle 
Mated Elements, Space Shuttle LPS, and Space Shuttle payloads, and near real-time mission 
support. 

Top-level functions for the SAIL are: 

1 . Full Orbiter ship set of flight configuration avionics. 

2. Sufficient mated element flight configuration avionics to check all interfaces. 

3. Subsystem to subsystem integration that make up the flight systems. 

4. Hardware/software flight system integration to verify mission capability. 

5. Fonnal test operations and failure reporting. 

6. Rigorous quality control and configuration management. 
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7. Electrical interfaces and physical space to accommodate payload testing. 

8. Located at JSC to use Government facilities. 

The SAIL objectives include: 

• Systems Verification 

- evaluation of subsystem interfaces 

- hardware and software integration 

- validation of vehicle ground checkout software 

- evaluation of redundancy management 

- determination of system limitations 

- mated element integration 

- launch processing system integration 

• Mission Verification 

- hardware and software capability 

- flight operating procedures such as in-flight maintenance 

• Operational Support 

- real-time mission support 

- payload integration 

- anomaly investigation 

The SAIL flight system (OV-095) configuration tracks and complies with the vehicle flight 
configuration. This includes the electrical power distribution and control; displays and controls; 
GN&C; data processing: mated elements [Space Shuttle Main Engine (SSME), Solid Rocket 
Booster (SRB, and External tank ET)]; payload interfaces; caution & warning; RMS; range 
safety: and Orbiter wire harness. The philosophy relative to non-avionics was anything that 
exercised flight software where it was not practical to have flight hardware would be simulated 
in the SAIL non- avionics simulation. For Ku-Band, the SAIL provided integration within the 
subsystem as well as subsystem-to-system level integration. Figure 5.3-3 shows the Space 
Shuttle test station, including the SAIL flight system. Figure 5.3-4 shows the middeck of the 
forward avionics bays. Figure 5.3-5 shows the aft avionics bays with two SSME controllers. 
Figure 5.3-6 shows the LPS. Figures 5.3-7 and 5.3-8 show the new/upgrade of the Test 
Operations Center and digital strip chart recorders. 
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Figure 5.3-3. Shuttle Test Station, including the SAIL Flight System 



Figure 5.3-4. Middeck of the Forward Avionics Bays 
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Figure 5.3-6. Launch Processing System 
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Figure 5.3-7. Test Operations Center 



Figure 5.3-8. Digital Strip Chart Recorders 


The overall verification process was a joint effort of civil service and contractor personnel. This 
effort was organized by engineering teams usually led at the subsystem level. These teams were 
part of the SAIL test team, as required, and participated from requirements definition through 
test procedure development, test execution, data analysis, and problem identification and 
correction/retest. Figure 5.3-9 illustrates the verification process flow. 
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Figure 5.3-9. SAIL Verification Process Flow 


The avionics verification process began when the software was released from the software 
production/reconfiguration facility and delivered to the SAIL. It was then integrated and closed- 
loop tested, with the avionics flight hardware in simulated mission sequences. Actual flight 
procedures/flight data files were integrated as appropriate into the SAIL Integrated Test and 
Checkout Procedures (ITCP). One purpose of the SAIL testing was to assure the avionics flight 
hardware and software would safely perform the upcoming mission. The process includes the 
verification of all hardware and software changes since the previous flight and the re-verification 
of the base software capabilities when new software Operational Increment (01) was released for 
integration and test in the SAIL. One of the key elements of the verification process was the 
audit trail and accountability for all hardware and software changes from requirements to flight 
readiness. This is implemented through avionics engineering teams who represent ascent, on- 
orbit, descent, avionics services, and data processing. An avionics flight system team also 
integrates across the other operations teams. The SAIL test requirements were coordinated 
through each of the avionics teams with a starting point of "what verification testing do you 
require in order to sign a flight readiness statement for your area of responsibility?" One part of 
the process was an evaluation of all hardware and software changes in the respective area of 
responsibility, and a definition of the type of test required to verify the respective system. The 
appropriate hardware changes/modifications were installed in the SAIL. These tests were not 
designed to be part of a test series on the actual flight vehicle. 
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Overall, SAIL provided the capability to identify problems and implement and test corrective 
action using an integration/verification laboratory rather than the utilizing an Orbiter. 
Approximately 300 software problems and approximately 50 hardware problems were identified 
as a result of integration and test in the SAIL prior to the first operational flight test. 

5.3.1 SAIL Summary 

In summary, the SAIL provided the capability to: 

1 . Verify the systems functional and compatibility of the integrated Orbiter avionics and the 
associated software. 

2. Verify the human/machine and avionics interfaces, integrated functions, avionics 
capability of the Orbiter, the mated elements (SSME, SRB, and ET) and the LPS. 

3. Verify the avionics interfaces between payloads and the Orbiter. 

4. Perform systems level investigations and verifications for mission unique software, 
mission functional requirements, operational anomalies, and effects of system design 
changes. 

5.4 CAIL Concept 

The CAIL concept is nearly identical to the SAIL approach. It is envisioned as a high fidelity, 
human-in-the-loop, and closed-loop flight simulation environment for the CEV avionics 
integration testing, which can execute mission scenario, ground and flight, nominal and off- 
nominal conditions. 

The Orion CAIL team identified the CAIL’s primary functions 1 : 

• Integration testing of the CEV avionics hardware and software and associated internal 
and external CEV interfaces. 

• CxP element-to-element integration testing as the CEV element. 

• Test and analysis operational support during CEV missions. 

The Orion CAIL team identified CAIL’s secondary functions : 

• Artifacts of a high fidelity flight-configuration test rig. 

• Engineering evaluation of proposed designs or subsystem/software changes. 


'Section 10.0, Reference 3 
2 Section 10.0, Reference 3 
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• Crew training and procedures checkout. 

The NESC team commended the Orion CAIL team for recognizing the need to set a priority of 
CAIL functions and for limiting the scope of new demands. Without these controls, the CAIL 
system would quickly exceed the available resources (e.g. cost and schedule). Although the 
outbriefs indicated there would be two production and one development/maintenance CAIL rigs, 
the Orion CAIL team noted the program plan includes three identical production rigs and no 
development/maintenance rig. The reason for this change was the projected increased non- 
recurring engineering cost of the development/maintenance rig. 

A key assumption made by the Orion CAIL team was that all avionics, software and GN&C 
subsystems delivered to CAIL will have been verified at the component and/or assembly level. 
This implies that there will be minimal development activities in the CAIL. To be more precise, 
the elements delivered to the CAIL will already have been operating at an entire system level and 
thus CAIL is not the first time the elements operate at a system level. Figure 5.4-1 indicates the 
CEV development flow is vertical (low- to high-level). 


SPACECRAFT LEVEL TEST 
(QUAL. NON-RECURRING TESTS AT SIT 



System / O&C 
Testing 


LEGE ND 

' CEV requirements verified 
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Figure 5.4-1. CEV Development Flow 
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The NESC team focused on the depicted flow of elements into the system verification [CAIL, 
Electronics Systems Test Laboratory (ESTL)] and the LM EDL. The NESC team was expecting 
a more iterative process as shown in the SAIL or evolving process as discussed in the Flat-Sat 
approach. Additionally, the EDL is positioned in the development flow as the Flat-Sat, and 
therefore could construe that it performs a similar function. However, it was noted by the Orion 
CAIL team that the EDL is not dedicated to the development process rather the purpose is 
defined by LM as “Develop and apply technology” and “Make Constellation successful” 3 . 
Clearly, the development function will make CxP successful, but the development and apply 
technology is more focused on future CxP elements and can compete for utilization time. The 
EDL is an “investment” into CxP by LM and contractor partners; essentially “LM builds it, 
NASA can use it”. It was not possible to determine the exact role the EDL will play in the flight 
avionics system development. It is important to note that there are four vehicle management 
computer test beds planned for avionics and software development and verification testing at the 
subsystem level. Thus, the Orion CAIL team has several options to fill the role of the 
development tool: dedicate the EDL, increase the fidelity of a vehicle management computer test 
beds, or decrease the fidelity of one of the CAIL systems. 

The requirement 4 of the CAIL portability was raised during CxP and NESC team discussions. 
Portability implies a level of modularity that does not exist in the flight article and appears 
contrary to the necessity to duplicate the flight system. When conflicting requirements exist, a 
priority must be established to guide the desired result. A review of the Spacecraft Test & 
Verification Facility System Requirements Document (CEV-T-082100) did not confirm a clear 
hierarchy. 

5.4.1 CAIL Summary 

The strategic approach of the CAIL is similar to the SAIL approach. Essentially, the approach is 
to create a tool of sufficient fidelity to enable a high fidelity, human-in-the-loop, closed-loop 
flight simulation environment for CEV avionics integration testing with the intent of reducing 
the risk of undiscovered safety discrepancies residing in the flight avionics system. Rigor in 
fidelity and configuration control required for verification of first mission and subsequent 
missions is cumbersome and not optimized for system development. A key assumption is the 
flight units delivered to the CAIL will be ready for system level verification and beyond 
development phase. The CAIL is a tool enabling formal flight readiness verification or pre- 
verification for formal verification on the flight article. 


3 

Exploration Development Lab (EDL) Overview Presentation (Slide 6), Lockheed Martin-owned and operated for Constellation Program and 

internal IR&D of Exploration Capabilities. 

4 

“Level A Functional Requirements, For the Crewed Exploration Vehicle (CEV) Avionics Integration Lab (CAIL)”, 3.3.3 Transition to 
Sustained Operations: The CAIL test system(s) shall be portable. 
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The development process from concept to flight system will utilize different tools as the design 
matures and the CAIL verification function must be aligned and coordinated with other planned 
end-to-end tools. It was not evident that the interaction of the different Orion development tools 
has been sufficiently coordinated. The Orion CAIL team recognized the coordination necessity 
and conducted a Technical Interchange Meeting (TIM) on the end-to-end testing in July 2007 
and a follow-on meeting that occurred in the spring of 2008. 

5.5 Software Test Beds 

During a development process a large portion of the time will be spent on software development, 
test and verification with the hardware. The overall development plan will directly influence the 
test bed fidelity and indirectly defines the scope and needs of the laboratory customers. For a 
software test bed there are a large number of users (simulation developers, hardware and 
software integration and test engineers, ground operations and ground application developers, 
mission operators, external hardware interface engineers, and system integration engineers). The 
conglomerate of users dictates a tightly coordinated delivery scheduling and user scheduling. 

5.5.1 An Example of Distributed Software Development Analysis, Evaluation, 

Integration, and Test 

Test beds can be supplied to external developers. The James Webb Space Telescope (JWST) 
exported project-owned test beds to the telescope instrument developers. The test beds consisted 
of commercial off-the-shelf (COTS) processors, high-fidelity hardware interfaces, the GSFC- 
developed Command and Data Handling (C&DH) flight software, the actual mission operations 
ground software, and an identical software development tool suite used at GSFC. The external 
developer is able to deliver flight-ready instrument software, instrument integration tests, and a 
set of mission procedures and the mission operations database. These deliverables are verified at 
their development site in the instrument operational environment. 

The instrument developers are using the test beds to support flight software development, 
environment test development, integration test development, system test procedures, and mission 
operations database and procedures. The exported test bed supported high-fidelity software 
interface verification with the C&DH, early high-fidelity subsystem hardware and software 
integration test development, mission operations database development, and mission operations 
process development. 

The GSFC developer is able to deliver C&DH software updates to the external developers for 
early subsystem integration and verification. A side effect of the exported test bed was the 
increased fidelity of communication between the external developers and GSFC. A shared 
knowledge of the test system developed due to the similarity of the development and test 
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environments. Shared knowledge of the JWST system developed due to the consistent system 
visibility. Solutions to shared issues were solved by involving all the developers. Expertise 
gained at one site was shared by the other sites. Late development at one site quickly "caught 
up" with the integration schedule through this information sharing, which in turn greatly reduced 
the learning-curve on the test beds. 

5.6 Comparison Summary of SAIL, CAIL, and Flat-Sat 

Figure 5.1-2 indicates the different purposes the Flat-Sat, SAIL and CAIL tools provide in 
relationship to the development cycle: 

• The Flat-Sat is a tool to mature a single mission concept into a flight avionics system and 
associated ground controller. The Flat-Sat fidelity progression enables rapid design 
maturation and earlier delivery enables more time on system with an opportunity to 
identify and correct discrepancies. 

• For the SAIL/CAIL tools, a high fidelity and rigorous controlled tool enables first 
mission and subsequent mission flight avionics verification with a high degree of 
confidence. The tools’ high fidelity enables rapid re-verification from mission to mission 
and provides programmatic risk management along with technical risk mitigation. 

Figure 5.1-2 indicates the different purposes the Flat-Sat, SAIL and CAIL tools provide in 
relationship to the development cycle. The Flat-Sat is a tool to mature a mission concept into a 
flight avionics system and associated ground controller. Then, the software development along 
with mission operations support follows. The Flat-Sat initial lower fidelity enables rapid 
maturity of design and lower system fidelity traded for earlier delivery enables more time on 
system and an opportunity to identify and correct discrepancies. Due to the single mission 
dictating that the fonnal verification process must occur on the flight article, there is no need for 
formal verification (and thus high fidelity) on the Flat-Sat tool. 

For the SAIL/CAIL tools, a high fidelity and rigorous controlled tool enables first mission and 
subsequent mission flight avionics verification with high degree of confidence. The tools’ high 
fidelity enables rapid re-verification from mission to mission and provides programmatic risk 
management along with technical risk mitigation. 

These tools are not exclusive; rather, they are complimentary in mitigating different risks 
encountered along the development from concept to operations. The NESC team noted that, 
based on their experience, the successful projects are keenly aware of their available tools’ 
abilities and limitations and can quickly adjust to the dynamically-changing risk priority and 
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hardware/software configurations of the development phase. Therefore, it is more important to 
have a complete tool set, than selecting any one individual tool. 

6.0 Comments from NESC Preliminary Briefing (March 17, 2008) 

It was observed by the NESC team that the requirement for “geometrically identical” in the 
CAIL was not addressed. This requirement sets a threshold on the risks that the CAIL must 
duplicate and thus provide the opportunity to discover these risks/discrepancies. To be 
quantitative of the benefit/cost of this risk threshold, it would require a statistical analysis of 
numerous past projects based on the individual projects’ risk, fidelity of their tools, and types of 
risks discovered. Unfortunately, this data is not known to exist in any collective format. Based 
on experience and knowledge of the development process, a general relationship between risk 
and system fidelity can be identified. At the highest level, or first order, the design is verified at 
a system level by ensuring the results of the different control loop interactions are as intended 
under nominal conditions and a functional representative system will be required. The functional 
representative system can be black boxes and simulators with no attempt at harness duplication 
and implementation specific nuances. At the second order, the performance around the 
individual control loops is verified under all operational conditions and thus an accurate 
representation of the implementation voltage variation and delays will be required. At this 
second level, only the single control loop requires flight-like assemblies and wires to duplicate 
the intentional implementation variances and the interacting systems can be simulated. At the 
third order, the unintentional interactions between the systems’ are verified and thus high fidelity 
hardware with geometrically identical configuration is required. At this lower level, the intent is 
to identify the subtle environmental induced interactions such as subsystem-to-subsystem 
thermal interactions, electromagnetic interactions and mechanical motion interactions. It is 
important to point out the first two levels are necessary to thoroughly characterize the nonnal 
system variations thus enabling the ability to identify the more subtle unintentional interactions. 
For Constellation, the optimal solution is a complete set of development test beds or tools that 
address the risks at various levels of the design. 

Another CxP/NESC team discussion item related to the philosophy of requiring all verification 
on the flight article or allowing verification on a high fidelity flight-like article. Clearly, the 
NASA philosophy of “test as you fly, fly as you test” holds true in this case and it is desirable to 
complete all verification on the flight article. Completing all verification on the flight article 
mitigates the class of risks that may be unique to that individual flight article. However, there 
are some tests that require specific inputs to close the control loop or conditions that require 
inducing failures or simulated failures. For the Space Shuttle's size and complexity, system level 
verification on the flight vehicle was not practical. In these cases, it is a balance of the risk of 
disassembling the flight article to implement the unique inputs versus the risk of the unknown 
difference between the flight article and the high fidelity flight-like test unit. There are certain 
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attitude control system tests where it is lower risk to verify on a high fidelity flight-like article. 
Programmatic risks (e.g. schedule and cost) or flight article availability must be weighed against 
the risk of not completing all verifications on the flight article. Other risks that must be 
evaluated include usage limits for the flight article and cost, schedule and complexity of vehicle- 
in-the-loop testing. The risks must be considered on a case by case basis. 
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7.0 Findings, Observations, and Recommendations 

7.1 Findings 

The following NESC team findings were identified: 

F-l. A coordinated, end-to-end, box to system level development and verification process was 
not evident for the proposed CA1L concept. 

F-2. The CAIL assumption is that the flight subsystems will be matured for the system level 
verification. 

• Assumes returning to the previous level is not required. 

F-3. The Flat-Sat and SAIL approaches are two different tools designed to mitigate different 
risks. 

• Flat-Sat is designed to develop a mission concept into a flight avionics system and 
associated ground controller. 

• SAIL is designed to validate the flight avionics system. 

• The approaches are complimentary in addressing both the system development 
risks and mission verification risks. 

7.2 Observations 

The following NESC team observations were identified: 

0-1. Based on the team’s experience, required/requested peak usage of an integration lab 
typically exceeds lab availability even in a multi-shift operation. 

0-2. Both the CAIL and the EDL material did not refer to an intentional effort to provide an 
engineering team knowledge transfer to ground operations and mission operations teams. 

0-3. Coordination of the verification process is essential to avoid missing key risks and to be 
prepared for schedule deviations. 

0-4. Partitioning of integration facility into subsystem integration labs allows for “cloning” 
and “export” to external developers. 

0-5. The CAIL and the SAIL differ in the fidelity of the propulsion hardware and software 
interface systems. 
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7.3 Recommendation 

The following NESC team recommendation was made: 

R-l. Define, document, and manage a detailed interface between the design and development 
(EDL and other integration labs) to the verification laboratory (CAIL). [FI, F2, 03] 

- Interfaces (flight software/hardware, schedule deviations, system discrepancies) 

- Typically defined and documented by CAIL PDR 


8.0 Definition of Terms 


Corrective Actions Changes to design processes, work instructions, workmanship practices, 
training, inspections, tests, procedures, specifications, drawings, tools, 
equipment, facilities, resources, or material that result in preventing, 
minimizing, or limiting the potential for recurrence of a problem. 

Development Process of functional testing, interface testing, open-loop testing and 

close-loop testing necessary to mature a concept to operation system 


Finding A conclusion based on facts established during the assessment/inspection 

by the investigating authority. 

Flight Avionics All electrical/electronic systems that will fly on the vehicle. 

System 


Ground Controller Ground electrical/electronic systems, tools, and procedures that enable the 
development and operation of the Flight Avionics System. 


Lessons Learned Knowledge or understanding gained by experience. The experience may 
be positive, as in a successful test or mission, or negative, as in a mishap 
or failure. A lesson must be significant in that it has real or assumed 
impact on operations; valid in that it is factually and technically correct; 
and applicable in that it identifies a specific design, process, or decision 
that reduces or limits the potential for failures and mishaps, or reinforces a 
positive result. 

Observation A significant factor established during this assessment that supports and 

influences the conclusions reached in the statement of Findings and 
Rec ommendations . 
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Problem The subject of the independent technical assessment/inspection. 

Recommendation An action identified by the assessment/inspection team to correct a root 

cause or deficiency identified during the investigation. The 
recommendations may be used by the responsible C/P/P/O in the 
preparation of a corrective action plan. 

Root Cause Along a chain of events leading to a mishap or close call, the first causal 

action or failure to act that could have been controlled systemically either 
by policy/practice/procedure or individual adherence to 
policy/practice/procedure. 

Verification Formal process of demonstrating a design is flight ready and is assumed to 

be interchangeable in the general sense to verification, validation and 
certification. 


9.0 List of Acronyms 


ALT 

AMD 

BFCS 

C&DH 

CA1L 

CDR 

COTS 

CxP 

EDL 

EOS 

ESMD 

ESTL 

ET 

ETU 

GN&C 

GSE 

GSFC 

l&T 

1TCP 

JPL 

JSC 


Approach and Landing Test 
Advanced Mirror Development 
Backup Flight Control System 
Communications and Data Handling 
CEV Avionics Integration Laboratory 
Critical Design Review 
Commercial-Off-The-Shelf 
Constellation Program 
Exploration Development Laboratory 
Earth Observing System 
Exploration Systems Mission Directorate 
Electronics Systems Test Laboratory 
External Ta nk 
Engineering Test Unit 
Guidance, Navigation, and Control 
Government Supplied Equipment 
Goddard Space Flight Center 
Integration and Test 

Integrated Test and Checkout Procedures 
Jet Propulsion Laboratory 
Johnson Space Center 
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JWST 

James Webb Space Telescope 

KSC 

Kennedy Space Center 

LM 

Lockheed Martin 

LPS 

Launch Processing System 

LRO 

Lunar Reconnaissance Orbiter 

MMES 

Marshall Mated Elements System 

NASA 

NASA Aeronautics and Space Administration 

NESC 

NASA Engineering and Safety Center 

NRB 

NESC Review Board 

OFT 

Orbital Flight Test 

OI 

Operational Increment 

OV 

Orbiter Vehicle 

PDR 

Preliminary Design Review 

RMS 

Remote Manipulator System 

RTLS 

Return To Launch Site 

SAIL 

Shuttle Avionics Integration Laboratory 

SDO 

Solar Dynamics Observatory 

SME 

Subject Matter Expert 

SRB 

Solid Rocket Booster 

SSME 

Space Shuttle Main Engine 

SSP 

Space Shuttle Program 

T&V 

Test & Verification 

TIM 

Technical Interchange Meeting 

TRMM 

Tropical Rainfall Measuring Mission 

VEST 

Vehicle Electrical System Test 

WMAP 

Wilkinson Microwave Anisotropy Probe 
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